Wide-field image input method and device

ABSTRACT

A wide-field image input device that joins a series of partial images obtained by continuously imaging an imaging object by an imaging device while the imaging device is manually scanned by a user, thereby generating a wide-field image, includes: a camera position/posture estimation unit for receiving a partial image series and calculating a change state of three-dimensional position and posture of the imaging device as time-series camera position/posture information; a deformation parameter estimation unit for calculating a geometrical deformation parameter between adjacent partial images from the camera position/posture information; an image joining unit for joining the partial images by using the geometrical deformation parameter so as to generate a wide-field image; and a camera scan error detection unit for detecting motion of the imaging device according to the camera position/posture information and outputting a camera scan error detection signal when an error is detected.

TECHNICAL FIELD

The present invention relates to a wide-field image input method anddevice for jointing a plurality of images that are taken, in order togenerate a wide-field image, when the number of pixels and the angle ofview in an imaging apparatus are smaller than those that are originallyrequired.

BACKGROUND ART

When a paper surface, such as magazines, newspapers, and books, isimaged by an imaging device, e.g. camera, there is a case where thenumber of pixels in an image pickup element device is insufficient toimage a desired range on the paper surface in a desired resolution. Inthis case, it is proposed that the camera images an object whilescanning the object as licking and a plurality of frame images that aretaken are jointed, thereby generating a wide-field image in highresolution.

Japanese Patent Laid-Open Application No. 11-298837 (JP, 11-298837A)proposes an image input device used in a case where images (partialimages) of adjacent scenes are joined to generate a piece of image. Theimage input device detects an overlapped region based on a motion vectorbetween taken partial images, and when the overlapped region has an areasufficient to calculate an amount of geometrical correction for joiningthe partial images without a sense of incompatibility, that effect isdisplayed on an indicator. A user can determine, from the indication,whether a joined image in a sufficient quality with inconspicuousboundaries between partial images is taken or not. Also, Japanese PatentLaid-Open Application No. 2004-96156 (JP, 2004-096156A) proposes animage input system for showing the resultant joined image to the user onan imaging site, in order to allow immediate confirmation whether imagescan be combined easily or not, i.e., whether failure or non-failure ofimaging, on the imaging site. A user can determine, from the presentedimage, whether the joined image in the sufficient quality withinconspicuous boundaries between partial images can be taken or not.

Now, when the plurality of partial images is joined to generate awide-field image, the number of samples per a unit length, i.e.,resolution, in each partial image varies in accordance with a distancebetween the camera and the object. For that reason, when the distancebetween the camera and the object varies, the wide-field image obtainedby joining has different resolutions in different portions.

Although the method is proposed in which the geometrical distortionbetween partial images caused by the tilt of the camera by hand shakesand variations in the distance are corrected and the partial images arethen joined, the joined image that is combined in this way includespartial blurry portions. Further, when the camera is tilted even in onepartial image, the image that has ubiquitously different resolutions istaken and resolution variations occur in the joined image. This problemoccurs more remarkably, when a relatively close object is imaged bymanual camera scan (i.e., the object is scanned while the camera ismoved), for example, the paper surface of newspapers or magazines isimaged, namely, when the object is close to the camera and is imaged ata wide-angle. In this description, the manual camera scan means that theobject is scanned while a camera is held by hands and is moved.

Specifically, the techniques disclosed in JP, 11-298837A and JP,2004-96156A described above target a use for panoramically imaging adistant view. In this use, though wobbling such as infinitesimal tiltscaused by hand shakes or the like occurs in the camera motion while thecamera is manually panned to take an image, the object is imaged withina level in which the object moves somewhat in parallel on the image.Therefore, hand shakes or the like have little effects on degradationsin the image quality of partial images that are taken and mosaic imagesthat are generated. However, when an object relatively close to thecamera is imaged, hand shakes or the like have a profound effect. Inother words, a wide-field image in partially low resolution withblurring is generated. Further, when the camera is excessively tilted,the image is taken in a manner that the paper surface has differentresolutions in different points in spite of one partial image.

The method is proposed in which, even if shakes occur in the cameramotion, a distortion parameter between partial images is estimated andthe partial images are accurately positioned, thereby generating amosaic image with inconspicuous joining points. However, no method iscarried out such that user's camera scan is guided so as not to generateresolution variations on the mosaic image. The mosaic image is an imagethat a character or the like in a printed-paper, such as newspapers andmagazines, is microscopically shown as mosaic.

Further, in order to solve these problems, when the camera scanningmethod is guided to the user so that a wide-field image can be generatedin a desired quality, it is difficult to provide how the position andthe orientation of the camera are corrected, for a user by aninstinctive method. As its reason, in the conventional panoramic imagingto take a distant scene as an object, as described above, variations inthe image caused by shakes, e.g., the rotation and the positional changeof the camera, are levels in that the object slightly rotates and movesin parallel. However, in a close scene, when the camera is tilted, theobject is distorted, and when the distance between the camera and theobject changes even slightly, the size is changed and imaged. The tiltand position change of the camera have a large effect on the resolutionof the partial image. It is difficult for the user to immediately graspwhether the camera has to be rotated or the position has to be movedwhen the camera scan is corrected.

FIG. 1 shows two images taken while scanning the same object. Assumingthat image 21 shown on the left side is a reference image and image 22shown on the right side is an image that is currently taken.Considerations are given to a case where it is necessary to correct theposition and posture of the camera so that these images can beaccurately positioned. In this case, it is difficult for the user toimmediately grasp whether the camera has to be rotated or the positionhas to be moved. As a method of directing the user to make alignment,there is a method in which the reference image that has been taken istranslucently superposed on an image that will be taken and the camerais moved to the position and posture to align them, like two examplesshown in FIG. 2. However, it is difficult for the user to immediatelygrasp which direction and how much the position and posture of thecamera are corrected only by referring to the superimposed image shownin FIG. 2. In FIG. 2, image 23 on the left side is an image in which aprevious frame image is superimposed on the image that is currentlytaken, and image 24 on the right side is an image in which the previousframe image is slightly shifted and superimposed on the image that iscurrently taken.

-   [Patent Document 1] JP, 11-298837A-   [Patent Document 2] JP, 2004-096156A-   [Non-Patent Document 1] Zelnik-Manor and Irani, “Multi-Frame    Estimation of Planar Motion,” IEEE Transactions on Pattern Analysis    and Machine Learning, Vol. 22, No. 10, (2000)

DISCLOSURE OF INVENTION Problems to be Solved by Invention

It is an object of the present invention to provide a wide-field imageinput device that can stably input a high-definition wide-field imageover the number of pixels in an image pickup element device and that canguide the manual scan for the imaging device by an intuitive method toenable a high-definition wide-field image to be input.

It is another object of the present invention to provide a wide-fieldimage input method that can stably input a high-definition wide-fieldimage over the number of pixels in an image pickup element device andthat can guide the manual scan for the imaging device by an intuitivemethod to enable a high-definition wide-field image to be input.

Means for Solving the Problem

The first object of the present invention is attained by a wide-fieldimage input device that joins a series of partial images obtained bycontinuously imaging an imaging object by an imaging device while theimaging device is manually scanned by a user, thereby generating awide-field image, the device including: camera position/postureestimation means for receiving a partial image series including theseries of partial images, and calculating the change state of thethree-dimensional position and posture of the imaging device astime-series camera position/posture information; deformation parameterestimation means for calculating a geometrical deformation parameterbetween adjacent partial images in the partial image series from thecamera position/posture information; image joining means for joining thepartial images while a deformation between the partial images iscorrected by using the geometrical deformation parameter so as togenerate a wide-field image; and camera scan error detection means fordetecting motion of the imaging device according to the cameraposition/posture information, and outputting a camera scan errordetection signal when an error is detected in the motion of the imagingdevice.

The first object of the present invention is also attained by awide-field image input device including: an imaging device forcontinuously imaging an imaging object while the imaging object ismanually scanned by a user, thereby obtaining a partial image seriesincluding a series of partial images; camera position/posture estimationmeans, based on the partial image series, for calculating the changestate of the three-dimensional position and posture of the imagingdevice as time-series camera position/posture information; deformationparameter estimation means for calculating a geometrical deformationparameter between adjacent partial images in the partial image seriesfrom the camera position/posture information; image joining means forjoining the partial images while a deformation between the partialimages is corrected by using the geometrical deformation parameter so asto generate a wide-field image; and camera scan error detection meansfor detecting motion of the imaging device according to the cameraposition/posture information and outputting a camera scan errordetection signal when an error is detected in the motion of the imagingdevice.

The second object of the present invention is attained by a wide-fieldimage input method that joins a series of partial images obtained bycontinuously imaging an imaging object by an imaging device while theimaging device is manually scanned by a user, thereby generating awide-field image, the method including the steps of: receiving a partialimage series including the series of partial images and calculating thechange state of the three-dimensional position and posture of theimaging device as time-series camera position/posture information;calculating a geometrical deformation parameter between adjacent partialimages in the partial image series from the camera position/postureinformation; joining the partial images while a deformation between thepartial images is corrected by using the geometrical deformationparameter so as to generate a wide-field image; and detecting motion ofthe imaging device according to the camera position/posture informationand outputting a camera scan error detection signal when an error isdetected in the motion of the imaging device.

According to the present invention, while the series of images is takenwith the imaging device, e.g., a camera, by the user, when there is apossibility in that the quality of images degrades caused by user'sscanning mistake such as that the camera is significantly tilted or thedistance from the imaging object is remarkably varied, that effect canbe notified to the user before such a scanning mistake is taken.Accordingly, the deterioration in the quality of mosaic images caused byuser's scanning mistake is reduced, and a wide-field image can beobtained in a stable image quality.

The wide-field image input device according to the present inventionalways monitors locus information and posture information of the camera,it can be grasped which direction and how much the position and postureof the camera are corrected when a scan error occurs. Accordingly, anappropriate instruction can be given to the user. Also, it can beintelligibly guided to the user how the object is scanned by imagingdevice, i.e., the camera in order to generate a high-definitionwide-field image effectively.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a view showing an example of partial images that are adjacentin time in a partial image series.

FIG. 2 is a view showing an example of guiding user's camera scan bysuperimposing two images.

FIG. 3 is a block diagram showing a configuration of a wide-field imageinput device according to a first embodiment of the present invention.

FIG. 4 is a block diagram showing a configuration of a wide-field imageinput device according to a second embodiment of the present invention.

FIG. 5 is a block diagram showing a configuration of another example ofthe wide-field image input device according to the second embodiment.

FIG. 6 is a view showing a display example for guiding a user by acamera position/posture indication unit.

FIG. 7 is a view showing a display example for guiding a user by acamera position/posture indication unit.

FIG. 8 is a view showing a display example for guiding a user by acamera position/posture indication unit.

FIG. 9 is a block diagram showing a configuration of a further exampleof a wide-field image input device according to the second embodiment.

FIG. 10 is a block diagram showing a configuration of a wide-field imageinput device according to a third embodiment of the present invention.

FIG. 11 is a block diagram showing a configuration of a wide-field imageinput device according to a fourth embodiment of the present invention.

EXPLANATION OF REFERENCE NUMERALS

1 Imaging device 2 Camera position/posture estimation unit 3 Deformationparameter estimation unit 4 Wide-field image storage unit 5 Imagejoining unit 6 Camera scan error detection unit 7 Cameraposition/posture indication unit 8 Absolute distance acquirement unit 9Blurring correction unit

BEST MODE FOR CARRYING OUT THE INVENTION First Embodiment

FIG. 3 shows a wide-field image input device according to the firstembodiment of the present invention. The wide-field image input devicejoins a plurality of partial images obtained by imaging device 1,thereby generating a wide-field image. In the present embodiment, a useruses imaging device 1 to image a scene as an object by manually scanningimaging device 1. Imaging device 1 continuously images the object at adesired angle of field, a desired position, and a desired posture andobtains a partial image series including a series of digitized partialimages. Imaging device 1 is embodied by, for example, a camera, inparticular, by an imaging unit in the camera.

The wide-field image input device is provided with cameraposition/posture estimation unit 2, deformation parameter estimationunit 3, wide-field image storage unit 4, image joining unit 5, andcamera scan error detection unit 6. Camera position/posture estimationunit 2 receives the digitized partial image series from imaging device 1and calculates the change state of the three-dimensional position andposture of the camera (imaging device 1) as time-series cameraposition/posture information, based on the received partial imageseries. Deformation parameter estimation unit 3 calculates a geometricaldeformation parameter between adjacent partial images in the partialimage series from the camera position/posture information or the partialimage series, or from both of them. Wide-field image storage unit 4stores the generated wide-field image.

Image joining unit 5 receives a series of geometrical deformationparameters from deformation parameter estimation unit 3 and receives thepartial image series supplied from imaging device 1 through cameraposition/posture estimation unit 2, joins the partial images whilecorrecting deformations between partial images to generate a wide-fieldimage, and outputs the generated wide-field image to wide-field imagestorage unit 4. The deformations between partial images mean changesbetween corresponding portions in adjacent partial images.

Camera scan error detection unit 6 receives the camera position/postureinformation through deformation parameter estimation unit 3 and detectsan error in the camera motion based on the received cameraposition/posture information, and provide an instruction to urge theuser to correct the camera scan as a camera scan error detection signalwhen the error is detected. The error in the camera motion includes, forexample, that the tilt angle of the camera is too large or the distancebetween the camera and the object is too large. Incidentally, in thepresent embodiment and the subsequent other embodiments, “camera” ispreferably configured by integrating imaging device 1 and the wide-fieldimage input device. In other words, imaging device 1 is preferably aconstitutional element of the wide-field image input device. Whenimaging device 1 and the wide-field image input device are integrated toconstitute a camera, the camera motion error is synonymous with themotion error in imaging device 1. The tilt and orientation of the cameraare represented by, for example, the tilt and orientation of an opticalaxis of a lens provided in imaging device 1. Additionally, according tothe present invention, imaging device 1 may be separated from thewide-field image input device in order to use a commercial camera or thelike as the imaging device.

Next, the operation of this wide-field image input device is explained.

The user scans the object by imaging device 1 in arbitrary motion suchthat the object is imaged as a whole. As a result, the partial imageseries is supplied from imaging device 1. In this case, imaging device 1may be any device, for example, devices such that to take movingpictures is started by the user operation of instructing the imagingstartup, such that images are continuously taken, and such that a seriesof still images are obtained whenever an imaging button is pushed.Imaging device 1 may output images other than those at the originalimaging timing to camera position/posture estimation unit 2 when imagesare continuously taken and still images are taken.

Camera position/posture estimation unit 2 receives a moving picture or astill image series supplied from imaging device 1 as the partial imageseries, determines how the object is imaged while being moved anddeformed on the image, and calculates the change in the position andposture of the camera relative to the object. A time-series of theposture and position of the camera is called camera position/postureinformation P={X₁, X₂, X₃, θ₁, θ₂, θ₃}. The fiducial point of the cameraposition/posture information may be any point in the three-dimensionalspace. For example, a coordinate on the joined image plane correspondingto the image center of the first frame may be used as the fiducialpoint. Then, the camera position/posture information may be representedby the optical central position of the camera (camera position) and theangle between the optical axis of the camera and each coordinate axis(camera posture) when the horizontal, vertical, and normal directionsfrom the fiducial point are respectively set as the coordinate axes.

Camera position/posture estimation unit 2 detects how arbitrary four ormore points on the taken image move in the following frame image,thereby calculating the change in the posture and position of thecamera. When a point in an image is given, a point that is to bedetected in the following frame image is called a corresponding point.For example, a pattern in a rectangular region around a point (x₁, y₁)as the center on the partial image is used as a template, the templateimage is searched on the following partial image, and a point (x₁′, y₁′)that is most matched is set as the corresponding point of point (x₁,y₁).

As one example, assuming that the above-mentioned process is executed todifferent four points on the image, four points ((x₁, y₁), (x₂, y₂),(x₃, y₃), (x₄, y₄)) on the image respectively move to ((x₁′, y₁′), (x₂′,y₂′), (x₃′, y₃′), (x₄′, y₄′)) on the following image. When a projectiontransformation matrix between images is represented by H, H isrepresented as followsH=[[h₁₁h₁₂h₁₃][h₂₁h₂₂h₂₃][h₃₁h₃₂h₃₃]].

When each element in projection transformation matrix H is normalized tosatisfy h₃₃=1, is arranged one-dimensionally, and is represented in avector, the matrix is obtained byH=[h ₁₁ h ₁₂ h ₁₃ h ₂₁ h ₂₂ h ₂₃ h ₃₁ h ₃₂1]^(T) =G ⁻¹ ·U.

Here,

$\begin{matrix}{{h_{33} = 1},} \\{U = \left\lbrack \begin{matrix}x_{1} & x_{2} & x_{3} & x_{4} & y_{1} & y_{2} & y_{3} & {\left. y_{4} \right\rbrack^{T},}\end{matrix} \right.} \\{G = {\begin{bmatrix}x_{1}^{\prime} & y_{1}^{\prime} & 1 & 0 & 0 & 0 & {{- x_{1}} \cdot x_{1}^{\prime}} & {{- x_{1}} \cdot y_{1}^{\prime}} \\x_{2}^{\prime} & y_{2}^{\prime} & 1 & 0 & 0 & 0 & {{- x_{2}} \cdot x_{2}^{\prime}} & {{- x_{2}} \cdot y_{2}^{\prime}} \\x_{3}^{\prime} & y_{3}^{\prime} & 1 & 0 & 0 & 0 & {{- x_{3}} \cdot x_{3}^{\prime}} & {{- x_{3}} \cdot y_{3}^{\prime}} \\x_{4}^{\prime} & y_{4}^{\prime} & 1 & 0 & 0 & 0 & {{- x_{4}} \cdot x_{4}^{\prime}} & {{- x_{4}} \cdot y_{4}^{\prime}} \\0 & 0 & 0 & x_{1}^{\prime} & y_{1}^{\prime} & 1 & {{- y_{1}} \cdot x_{1}^{\prime}} & {{- y_{1}} \cdot y_{1}^{\prime}} \\0 & 0 & 0 & x_{2}^{\prime} & y_{2}^{\prime} & 1 & {{- y_{2}} \cdot x_{2}^{\prime}} & {{- y_{2}} \cdot y_{2}^{\prime}} \\0 & 0 & 0 & x_{3}^{\prime} & y_{3}^{\prime} & 1 & {{- y_{3}} \cdot x_{3}^{\prime}} & {{- y_{3}} \cdot y_{3}^{\prime}} \\0 & 0 & 0 & x_{4}^{\prime} & y_{4}^{\prime} & 1 & {{- y_{4}} \cdot x_{4}^{\prime}} & {{- y_{4}} \cdot y_{4}^{\prime}}\end{bmatrix}.}}\end{matrix}$Camera position/posture estimation unit 2 can obtain H by the leastsquare method when the number of corresponding points are large on theimage. External parameter matrix M₁ of the camera at this time isobtained byM ₁ =M ₀ ·H ₀₁.In this equation, M₀ corresponds to the camera position/postureinformation of the first frame, and when measurement is difficult,P₀=[0, 0, 1, 0, 0, 0] or the like may be used instead of M₀. H₀₁ isprojection transformation matrix H between images of the first frame andthe following frame. Also, external parameter matrix M of an arbitraryframe is obtained byM=M ₀ ·H.In this situation, assuming that matrix H is a projection transformationmatrix between images of the first frame and that frame. In this case,using elements in camera position/posture information P, M isrepresented by

$M = {\begin{bmatrix}{{\cos\;{\theta_{1} \cdot \cos}\;\theta_{3}} + {\sin\;{\theta_{1} \cdot \sin}\;{\theta_{2} \cdot \sin}\;\theta_{3}}} & {\sin\;{\theta_{1} \cdot \cos}\;\theta_{2}} & X_{1} \\{{- \sin}\;{\theta_{1} \cdot \cos}\;\theta_{3}} & {\cos\;{\theta_{1} \cdot \cos}\;\theta_{2}} & X_{2} \\{\cos\;{\theta_{2} \cdot \sin}\;\theta_{3}} & {{- \sin}\;\theta_{2}} & X_{3}\end{bmatrix}.}$Therefore, elements in P can be obtained, as appropriate. Cameraposition/posture estimation unit 2 repeatedly calculates cameraposition/posture information P, which can be obtained, e.g., by theabove-mentioned method, whenever a partial image is supplied, andoutputs camera position/posture information P to deformation parameterestimation unit 3 and camera scan error detection unit 6 whenever it iscalculated.

Deformation parameter estimation unit 3 receives the partial image orthe camera position/posture information, or both of them as an input,and calculate a geometrical deformation parameter for correcting thegeometrical deformation in the partial images in order to join thepartial images without feeling abnormality. Assuming that the object isa plane, when the partial image series is input, eight parameters areestimated while the geometrical deformation between the partial imagesis regarded as the plane projection transformation. When the plane onwhich the joined image is generated is known, it may be calculated asthe projection transformation parameter between the joined plane and theplane. By observing images of several frames, the orientation of theobject plane can be known.

When the camera position/posture information is input, similarly, theplane projection transformation parameter to the joined image plane canbe calculated easily. When the camera position/posture information andthe partial image are available, deformation parameter estimation unit 3may apply the plane projection transformation parameter obtained fromthe camera position/posture information to the partial image and make acalculation to improve the accuracy of the plane projectiontransformation parameter in which errors on the joined image plane arecorrected. Alternatively, projection transformation matrix H obtained inthe process of calculating camera position/posture information P incamera position/posture estimation unit 2 may be used as the geometricaltransformation parameter, as it is.

Image joining unit 5 applies the geometrical deformation parametersupplied from deformation parameter estimation unit 3 to the partialimage supplied from imaging device 1 through camera position/postureestimation unit 2 and writes the resultant image into a predeterminedposition as a wide-field image so as to execute joining. At this time,the blending process may be applied, using the known mixture ratio, tooverlapped portions between the wide-field image that is completelygenerated and the partial image that is supplied while beinggeometrically deformed.

Camera scan error detection unit 6 always monitors the cameraposition/posture information supplied from camera position/postureestimation unit 2. When the camera moves away from the object fartherthan a preset value, compared with the initial camera position andposture, or when the pan angle and the tilt angle are larger than presetvalues, that is regarded as a camera scan mistake and a camera scanerror detection signal is output. Notification is given to the user inaccordance with output of the camera scan error detection signal. Forexample, when variations in the resolution of the joined image are seton the order of 10 percent of the maximum resolution, the tolerance forvariations in the distance between the camera and the object is set suchthat the ratio with the distance from the initial frame is within 1.1,and when the angle of field is 20 degrees, the tolerance for variationsin the angel is within approximately 22 degrees. Camera scan errordetection unit 6 outputs the camera scan error detection signal when itis detected that the variations in the distance and the variations inthe angle exceed the tolerances. Actually, when the distance ratio orthe angle reaches the upper limit of the above-mentioned tolerance, thevariations in the resolution reach the limit of the tolerance at thattime. Therefore, notification may be given before reaching the upperlimit. For example, when warning is given at 70 percent of the variationamount, i.e., 70 percent of the tolerance, concerning both the distanceratio and the angle, the thresholds thereof are 1.07 and 15.4 degrees.

Also, joining the images using the partial image series or the cameraposition/posture information input from imaging device 1 is previouslysuspended, and the joining may be started when a predetermined timeelapses or when it is confirmed that the accuracy of the orientation ofthe object plane can be obtained.

According to the wide-field image input device of the first embodiment,it is possible to inform the user of the deterioration in the quality ofmosaic images caused by user's mistake on the camera scanning such asthat the camera is remarkably tilted or the distance from the objectchanges significantly before the scan mistake apt to cause thedeterioration is taken.

Second Embodiment

FIG. 4 shows a wide-field image input device according to the secondembodiment of the present invention. The wide-field image input deviceshown in FIG. 4 is configured by adding camera position/postureindication unit 7 to the wide-field image input device according to thefirst embodiment. Camera position/posture indication unit 7 receives thecamera scan error signal from camera scan error detection unit 6 and thecamera position/posture information from camera position/postureestimation unit 2. Preferably, the camera scan error signal is camerascan error information including information which scan error occurs.

Camera position/posture indication unit 7 performs one or more of thefollowing four processes (a) to (d) of:

(a) visually indicating the change in the relative position between thecamera and the imaging object or the change in the camera posture, orthe change in the relative position and the change in the cameraposture;

(b) displaying warning when the relative position and/or the posture arein levels of having effects on the deterioration in the quality ofimages;

(c) displaying a correction amount by a gauge or an arrow to indicatewhich direction and angle and how much the camera has to be corrected,so as to take image appropriately; and

(d) indicating the change amount of the relative position and/or thechange amount of the posture by a tone at a pre-specified pitch.

Incidentally, when the camera scan error detection signal (camera scanerror information) is not used, camera position/posture indication unit7 may not receive the information from camera scan error detection unit6, as shown in FIG. 5.

Next, the operation of the wide-field image input device according tothe second embodiment is explained. As an example of the case of (a)mentioned above, a display (not shown), e.g., a finder or a display unit(monitor) arranged in the digital still camera displays the orientationand posture of the camera based on the camera position/postureinformation supplied from camera position/posture estimation unit 7. Atthis time, for example, only the camera posture information may bedisplayed. In this case, as an example shown in FIG. 6 illustrates, whenrectangle 202 corresponding to the aspect ratio of the frame, a markshowing optical center 201, and each line segment for linking eachvertex of the rectangle and the optical center are drawn, the user cangrasp the posture information easily. Incidentally, FIG. 6 shows theexample of the camera apparatus, e.g., the digital camera of which theshape of the plane is rectangle.

When the camera is oriented in the vertical downward direction, {θ₁=0,θ₂=0, θ₃=0}, the mark showing optical center 201 is displayed at thecenter of rectangle 202. When θ is not zero, respective vertexes 203,204, 205, 206 of rectangle 202 are shifted and displayed in accordancewith the magnitude of θ while optical center 201 is fixed, as an exampleshown in FIG. 7 illustrates. Since θ₃ has no effect on the resolution inthe image, only the vertex positions may be shifted asU_(n)=U_(n+k)·θ_(n) (n=1, 2) in accordance with the magnitudes of θ₁,θ₂. Alternatively, concerning matrix M for representing the cameraposition/posture information including θ₃ and distance X₃ to the object,the inverse matrix thereof may be obtained while X₁ and X₂ are set to 0,and the vertex positions may be shifted to positions calculated bymultiplying vector c(u₁, u₂, 1) obtained from the coordinatescorresponding to the respective vertexes and the obtained inversematrix. In this description, c is a parameter for controlling the scaleof the rectangle display.

When the camera scan error detection information is received from camerascan error detection unit 6, the error detection information isdisplayed on the display screen, i.e., the finder or the display unit,or is indicated to the user by the tone, as cases (b) and (d) mentionedabove, thereby easily grasping which direction the camera posture has tobe corrected at that time. For example, the user corrects the cameraposture in the direction such that optical center mark 202 is shifted tothe center of rectangle 202, in order to eliminate the indication of thecamera scan error, and performs the camera scan, in the above-mentionedexample.

Further, as one example of case (c) mentioned above, when partial imagesare taken without portions that are mutually overlapped because thecamera scan is too fast, the direction of the frame that is finallytaken while being overlapped, relative to the current frame position, isdisplayed by arrow 207 on the display, as shown in FIG. 8. Also, whenthe wide-field image is generated by the still image taking, the cameraposition/posture estimation is always performed during the period afterthe previous partial image is taken until the following partial image istaken. In this case, the distance from optical center mark 201 to tip208 of the arrow may be changed and displayed in accordance withparallel movement amount X₁, X₂ from the frame that is finally takenwhile being correctly overlapped. In this way, by displaying the lengththat is changed, the user can easily understand how much the camera scanhas to be returned in the previous scanning direction. Also, when theposture correction information of rectangle 202 is combined, the usereasily grasps instinctively whether the posture has to be corrected orthe position has to be corrected.

When the camera scan error is not visually indicated to the user, thepitch of the tone is changed in accordance with the level of thedeviation of the camera position/posture information from the idealstate, as case (d) mentioned above. For example, the frequency of thetone may be made higher or lower in proportion to Euclidean distance ofsix-dimensional vector of the camera position/posture information. Thismay be combined with the indication aspects of (a) to (c) mentionedabove. Incidentally, for example, the tone is emitted from a soundoutput unit built in the device to which the present invention isapplied.

Further, the device may be configured such that, when overlap existsbetween partial images and no camera scan error is informed, an image isautomatically taken even if the user gives no imaging instruction toimaging device 1. For example, imaging device 1 may receive thegeometrical deformation parameter from deformation parameter estimationunit 3 and may receive the camera scan error detection information fromcamera scan error detection unit 6, as shown in FIG. 9. Specifically,when imaging device 1 confirms that the deformation amount of the objectbetween partial images is within the preset value, by the geometricalparameter and no camera scan error detection information is fed fromcamera scan error detection unit 6, images are continuously takenautomatically even if the user inputs no imaging instruction. Here,deformation parameter estimation unit 3 does not output the geometricaldeformation parameter to imaging device 1, but an imaging availablesignal may be supplied to imaging device 1 when it is confirmed that thedeformation amount of the object between the partial images is withinthe preset value by the geometrical deformation parameter. In this case,when the imaging available signal is supplied and no camera scan errordetection information is supplied, imaging device 1 continuously takesimages automatically even if the user inputs no imaging instruction.

With this arrangement, the user is not required to pay attentions to thetiming for imaging and can take a wide-field image while paying onlyattentions to that the camera scan is correctly performed. Incidentally,the imaging instruction, for example, is to push a shutter buttonarranged in the camera to which the present invention is applied. Amicrocomputer or the like installed in the camera makes imaging device 1take an image in accordance with pushing of the shutter button.

Third Embodiment

FIG. 10 shows a wide-field image input device according to the thirdembodiment of the present invention. This wide-field image input deviceis configured by adding absolute distance acquirement unit 8 to thewide-field image input device according to the first or secondembodiment. Absolute distance acquirement unit 8 measures the distancebetween the camera and the object and outputs distance informationshowing the measured distance to camera scan error detection unit 6.Absolute distance acquirement unit 8 like this is embodied by, forexample, a distance sensor which is able to measure the distance betweenthe camera and the object. The configuration shown in FIG. 10 is thatabsolute distance acquirement unit 8 is added to the wide-field imageinput device according to the second embodiment, however, absolutedistance acquirement unit 8 may be added to the wide-field image inputdevice according to the first embodiment shown in FIG. 3.

Also, according to the third embodiment, camera scan error detectionunit 6 monitors whether imaging over the predetermined samplingfrequency or predetermined resolution (number of pixels) is performed onthe object surface or not, and outputs the camera scan error informationto camera position/posture indication unit 7 when below.

Next, the operation of the wide-field image input device according tothe third embodiment is explained.

In the above-mentioned first and second embodiments, when it isdifficult to measure camera position/posture information P of the firstframe in the partial image series, element X₃ in P₀ is set to 1, as oneexample. However, in the third embodiment, X₃ is measured as an absolutedistance by absolute distance acquirement unit 8. Therefore, it ispossible to measure the length on the object surface and how many pixelscorrespond to the length on the image. In other words, the resolution ofthe partial image can be judged in accordance with the distanceinformation. The camera scan error is determined when the number ofpixels in a unit length is below a predetermined threshold, and can beinformed to the user.

As well known, the resolution in images is represented by the number ofdots per one inch, i.e., 25.4 mm, and “dpi” is a unit for representingthe number of dots per one inch.

For instance, an operation example is shown when a necessary resolutionis specified to 400 dpi as a predetermined value. Distance d on theobject surface between two points corresponding to A₁=(x₁, y₁, 1),A₂=(x₂, y₂, 1) on the same partial image is obtained with externalparameter matrix M corresponding to P={X₁, X₂, X₃, θ₁, θ₂, θ₃} byd=∥M ⁻¹ A ₂ −M ⁻¹ A ₁∥.Here, ∥·∥ represents the norm. Assuming that in three-dimensionalvectors M⁻¹A₂ and M⁻¹A₁, the first and second elements are normalized bythe third element. Also, assuming that distance d and coordinate valuesof x₁, x₂, y₁, y₂ are described using mm (millimeter) as a unit.

Then, the number of pixels per one inch on the line linking A₁ and A₂ onthe partial image is represented by

${\frac{\sqrt{\left( {x_{1} - x_{2}} \right)^{2} + \left( {y_{1} - y_{2}} \right)^{2}}}{d} \times 25.4}.$The camera scan has an error when it is below 400. The calculation likethis is applied to each partial image or the frame image between imagingand imaging, thereby detecting a scan error.

Incidentally, assuming that 400 dpi is specified, similarly to the firstembodiment, no notification is given when it is below 400 dpi, butnotification is given when it is below a value which is larger than thespecified value by a preset value (for example, 440), thereby preventingfailure of imaging. The operations of the other processing units aresimilar to those of the other embodiments.

Fourth Embodiment

FIG. 11 shows a wide-field image input device according to the fourthembodiment of the present invention. The wide-field image input deviceis configured by adding blurring correction unit 9 to the wide-fieldimage input device according to the first embodiment. Blurringcorrection unit 9 receives the camera position/posture informationsupplied from camera position/posture estimation unit 2 as an input,estimates a blurring function included in the partial image taken byimaging device 1, performs the blurring correction to the partial imagesupplied from imaging device 1 in accordance with the estimated blurringfunction, and outputs it to image joining unit 5.

Next, the operation of this wide-field image input device is explained.

Blurring correction unit 9 measures the moving speed of pixels on theimage with projection transformation matrix H supplied from deformationparameter estimation unit 3. For example, it is calculated by H that onepoint on the image shifts to which coordinate in the following frame,and the distance between them is set to two-dimensional vector D=(D_(x),D_(y)). When the time interval used to take two frames is set as T, thespeed is represented by D/T. For example, assuming two-dimensionalnormal distribution N(0, Σ), the blurring function may be

${\sum{= \begin{bmatrix}{{kS}\left( {D_{x}/T} \right)}^{2} & {{{kS}\left( {D_{x}/T} \right)}\left( {D_{y}/T} \right)} \\{{{kS}\left( {D_{x}/T} \right)}\left( {D_{y}/T} \right)} & {{kS}\left( {D_{y}/T} \right)}^{2}\end{bmatrix}}},$where kS is a proportionality constant. When the shutter speed of thecamera into which the wide-field image input method of the thirdembodiment is incorporated is known, the shutter speed may be used as Sand may be reflected on the calculation of Σ.

As an reconstruction filter for removing blurring, for example,K=N/(|N|2+Γ)is used. Γ is a preset constant. All the partial image series input fromimaging device 1 passes through the reconstruction filter, and then isfed to image joining unit 5. The processes in the other processing unitsare similar to those of the other embodiments.

As described above, the preferred embodiments of the present inventionare explained, and the following examples can be considered as modifiedexamples of these embodiments. For example, in the wide-field imageinput device according to each of the above-mentioned embodiments, whenthe initial camera posture is unknown, image joining unit 5 may beconfigured to perform a process of obtaining the posture informationfrom the series of partial images, obtaining an object orientationsuitable for ex post facto viewing the object, and adjusting theorientation of the joined image in accordance with the objectorientation.

Also, in the wide-field image input device according to the first,second, and third embodiments, posture information {θ₁, θ₂, θ₃} out ofcamera position/posture information P₀ corresponding to the first framecan be calculated after taking two or more frames by the method proposedby Zelnik-Manor and Irani (Zelnik-Manor and Irani, “Multi-FrameEstimation of Planar Motion,” IEEE Transactions on Pattern Analysis andMachine Learning, Vol. 22, No. 10, (2000)). In the first frame, P₀=[0,0, 1, 0, 0, 0] is assumed, P₀ is set to [0, 0, 1, θ₁, θ₂, θ₃] afterimaging several frames, and is corrected by corresponding externalparameter matrix M₀ to perform display, thereby monitoring the posturefurther correctly.

Also, in imaging the partial image series, P₀=[0, 0, 1, 0, 0, 0] is set,and P₀ is corrected to [0, 0, 1, θ₁, θ₂, θ₃] after imaging all frames,thereby generating the wide-field image in which the tilt of the camerain the first frame is corrected. Also, the joining process may be notstarted until the initial camera posture is calculated with a presetaccuracy, and the imaging for joining may be started after determiningthe object orientation suitable for ex post facto viewing the objectbased on the obtained posture information.

Further, when the process by camera position/posture estimation unit 2ends in failure, image joining unit 5 suspends joined images before thefailure and starts to generate another new joined image. When the finishof the acquirement operation for the final partial image is notifiedfrom imaging device 1, mutual joining process may be applied to all thejoined images that are suspended before the current time, therebyrestoring the joining processes that end in failure before the mutualjoining process.

Incidentally, each of the above-mentioned embodiments can be applied toportable terminal equipment having an imaging function, such as a videocamera; a digital still camera; a mobile telephone, PHS (PersonalHandyphone System), a personal computer and PDA (Personal DataAssistance, Personal Digital Assistants) which are provided with acamera; or to an imaging device that can control an imaging range, suchas an active camera and a robot.

For application to the portable terminal equipment or the imagingdevice, mentioned above, each processing unit except imaging device 1and the wide-field image storage unit can be carried out by amicrocomputer, specifically, by CPU that operates in accordance withprograms for carrying out respective functions. The respective functionsinclude a camera position/posture estimation function for carrying outcamera position/posture estimation unit 2, a deformation parameterestimation function for carrying out deformation parameter estimationunit 3, an image joining function for carrying out image joining unit 5,a camera scan error detection function for carrying out camera scanerror detection unit 6, a camera position/posture indication functionfor carrying out camera position/posture indication unit 7, an absolutedistance acquirement function for carrying out absolute distanceacquirement unit 8 together with a distance sensor and the like, and ablurring correction function for carrying out blurring correction unit9.

Therefore, the present invention includes such a program, a storagemedium and a program product stored with the program in the scope of thepresent invention.

INDUSTRIAL APPLICABILITY

The present invention can be applied to the above-mentioned devices,such as a video camera and a digital still camera, and to a systemincluding the devices.

1. A wide-field image input device that joins a series of partial imagesobtained by continuously imaging an imaging object by an imaging devicewhile the imaging device is manually scanned by a user, therebygenerating a wide-field image, the device comprising: cameraposition/posture estimation means for receiving a partial image seriesincluding said series of partial images, and calculating a change stateof three-dimensional position and posture of said imaging device astime-series camera position/posture information; deformation parameterestimation means for calculating a geometrical deformation parameterbetween adjacent partial images in said partial image series from saidcamera position/posture information; image joining means for joiningsaid partial images while a deformation between said partial images iscorrected by using said geometrical deformation parameter so as togenerate a wide-field image; and camera scan error detection means fordetecting motion of said imaging device according to said cameraposition/posture information, and outputting a camera scan errordetection signal when an error is detected in the motion of said imagingdevice.
 2. The device according to claim 1, further comprising cameraposition/posture indication means, based on said camera position/postureinformation from said camera position/posture estimation means, forperforming one or more of the processes consisting of: (a) visuallyindicating a change in a relative position between said imaging deviceand the imaging object or a change of posture of said imaging device, orthe change in the relative position and the change of posture; (b)displaying warning when said relative position and/or said posture arein levels of having effects on deterioration in a quality of images; (c)visually displaying a correction amount to indicate which direction andangle and how much said imaging device is corrected for desirableimaging; and (d) indicating a change amount of said relative position ora change amount of the posture, or both by a tone at a pitch.
 3. Thedevice according to claim 1 or 2, further comprising: absolute distanceacquirement means for measuring a distance from said imaging object andoutputting distance information to said camera scan error detectionmeans, wherein said camera scan error detection means is configures soas to judge a resolution of said partial image in accordance with saiddistance information, and to output said camera scan error signal to thecamera position/posture indication means when the resolutions does notreach a predetermined value.
 4. The device according to claim 3, whereinsaid image joining means is configured, when a initial camera posture isunknown, to obtain posture information from said partial image series,to obtain an object orientation suitable for ex post facto viewing anobject, and to adjust an orientation of a joined image in accordancewith the object orientation that is obtained.
 5. The device according toclaim 1, further comprising blurring correction means for estimating ablurring function included in a partial image taken by said imagingdevice based on said camera position/posture information, and forcorrecting said partial image supplied from said imaging device inaccordance with said blurring function that is estimated to output thepartial image to the image joining means.
 6. A wide-field image inputdevice comprising: an imaging device for continuously imaging an imagingobject while the imaging device is manually scanned by a user, and forobtaining a partial image series including a series of partial images;camera position/posture estimation means, based on the partial imageseries, for calculating a change state of three-dimensional position andposture of said imaging device as time-series camera position/postureinformation; deformation parameter estimation means for calculating ageometrical deformation parameter between adjacent partial images insaid partial image series from said camera position/posture information;image joining means for joining said partial images while a deformationbetween said partial images is corrected by using said geometricaldeformation parameter so as to generate a wide-field image; and camerascan error detection means for detecting motion of said imaging deviceaccording to said camera position/posture information and outputting acamera scan error detection signal when an error is detected in themotion of said imaging device.
 7. The device according to claim 6,further comprising camera position/posture indication means, based onsaid camera position/posture information from said cameraposition/posture estimation means, for performing one or more of theprocesses consisting of: (a) visually indicating a change in a relativeposition between said imaging device and the imaging object or a changeof posture of said imaging device, or the change in the relativeposition and the change of posture; (b) displaying warning when saidrelative position and/or said posture are in levels of having effects ondeterioration in a quality of images; (c) visually displaying acorrection amount to indicate which direction and angle and how muchsaid imaging device is corrected for desirable imaging; and (d)indicating a change amount of said relative position or a change amountof the posture, or both by a tone at a pitch.
 8. The device according toclaim 6 or 7, wherein said deformation parameter estimation meansoutputs an imaging available signal to said imaging device when adeformation amount between partial images is within a predeterminedvalue, wherein said camera scan error detection means outputs saidcamera scan error detection signal to said imaging device, and whereinsaid imaging device is configured so as to perform imaging automaticallywhen said imaging available signal is received and said camera scanerror signal is not supplied.
 9. The device according to claim 8,wherein said image joining means is configured, when a initial cameraposture is unknown, to obtain posture information from said partialimage series, to obtain an object orientation suitable for ex post factoviewing an object, and to adjust an orientation of a joined image inaccordance with the object orientation that is obtained.
 10. The deviceaccording to claim 6 or 7, further comprising: absolute distanceacquirement means for measuring a distance from said imaging object andoutputting distance information to said camera scan error detectionmeans, wherein said camera scan error detection means is configured soas to judge a resolution of said partial image in accordance with saiddistance information, and to output said camera scan error signal to thecamera position/posture indication means when the resolutions does notreach a predetermined value.
 11. The device according to claim 10,wherein said image joining means is configured, when a initial cameraposture is unknown, to obtain posture information from said partialimage series, to obtain an object orientation suitable for ex post factoviewing an object, and to adjust an orientation of a joined image inaccordance with the object orientation that is obtained.
 12. The deviceaccording to claim 6, further comprising blurring correction means forestimating a blurring function included in a partial image taken by saidimaging device based on said camera position/posture information, andfor correcting said partial image supplied from said imaging device inaccordance with said blurring function that is estimated to output thepartial image to the image joining means.
 13. The device according toany one of claims 1, 2, 5-7, and 12, wherein said image joining means isconfigured, when a initial camera posture is unknown, to obtain postureinformation from said partial image series, to obtain an objectorientation suitable for ex post facto viewing an object, and to adjustan orientation of a joined image in accordance with the objectorientation that is obtained.
 14. A wide-field image input method thatjoins a series of partial images obtained by continuously imaging animaging object by an imaging device while the imaging device is manuallyscanned by a user, thereby generating a wide-field image, the methodcomprising the steps of: receiving a partial image series including saidseries of partial images, and calculating a change state ofthree-dimensional position and posture of said imaging device astime-series camera position/posture information; calculating ageometrical deformation parameter between adjacent partial images insaid partial image series from said camera position/posture information;joining said partial images while a deformation between the partialimages is corrected by using said geometrical deformation parameter soas to generate a wide-field image; and detecting motion of said imagingdevice according to said camera position/posture information, andoutputting a camera scan error detection signal when an error isdetected in the motion of said imaging device.